Papers with instruction-tuned language models
SymPyBench: A Dynamic Benchmark for Scientific Reasoning with Executable Python Code (2026.eacl-industry)
Copied to clipboard
| Challenge: | Existing benchmarks do not capture the complexity of structured, step-by-step reasoning essential in physics and related domains. |
| Approach: | They propose a large-scale synthetic benchmark of 15K university-level physics problems . they use structured, step-by-step reasoning and executable Python code to produce the ground-truth solution. |
| Outcome: | The proposed model is based on a set of 15K university-level physics problems with three question types. |
Leveraging Contextual Information for Effective Entity Salience Detection (2024.findings-naacl)
Copied to clipboard
Rajarshi Bhowmik, Marco Ponza, Atharva Tendle, Anant Gupta, Rebecca Jiang, Xingyu Lu, Qian Zhao, Daniel Preotiuc-Pietro
| Challenge: | Prior work on salient entity detection focused on machine learning models that require heavy feature engineering. |
| Approach: | They propose to fine-tune medium-sized language models with a cross-encoder style architecture to achieve significant performance gains over feature engineering approaches. |
| Outcome: | The proposed model fine-tunes medium-sized pre-trained language models with a cross-encoder style architecture yields substantial performance gains over feature engineering approaches. |
Measuring and Mitigating Shortcut Reliance in Language Models with Probe-Based Representation Entanglement (2026.acl-srw)
Copied to clipboard
| Challenge: | Shortcut learning remains a major obstacle to robust NLP systems. |
| Approach: | They propose to fine-tune Gemma 3 1B Instruct and Llama 3.2 1B on two synthetic sentiment shortcuts in SST-2 and one natural shortcut in MNLI based on lexical overlap. |
| Outcome: | The proposed model improves on two synthetic sentiment shortcuts and one natural shortcut in MNLI with a 99% shortcut ratio, while Gemma drops from 91.8% to 60.2%. |
ARQA: A Benchmark for Grounded Table–Text QA in Enterprise Annual Reports (2026.eacl-industry)
Copied to clipboard
| Challenge: | Existing QA benchmarks focus on retrieval or single-modality reasoning . annual reports are a company's definitive record of performance . |
| Approach: | They propose an annual report QA benchmark that compares QAs with lookups, arithmetics, and insights. |
| Outcome: | The proposed benchmarks show strong factual retrieval but persistent weaknesses in grounded arithmetic and causal reasoning. |
Unsupervised Task Graph Generation from Instructional Video Transcripts (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to task graph generation use instruction-tuned language models to generate accurate task graphs. |
| Approach: | They propose an unsupervised approach that combines the reasoning capabilities of instruction-tuned language models with clustering and ranking components to generate accurate task graphs. |
| Outcome: | The proposed approach generates more accurate task graphs than a supervised learning approach on tasks from the ProceL and CrossTask datasets. |
QA-NatVer: Question Answering for Natural Logic-based Fact Verification (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent work has focused on natural logic, which operates directly on natural language by capturing the semantic relation of spans between an aligned claim and its evidence via set-theoretic operators. |
| Approach: | They propose to use question answering to predict natural logic operators using generalization capabilities of instruction-tuned language models. |
| Outcome: | The proposed approach outperforms the best baseline on a Danish verification dataset by 4.3 accuracy points. |
Patches of Nonlinearity: Instruction Vectors in Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Despite the success of instruction-tuned language models, little is known about how they process instructions internally. |
| Approach: | They propose a method to localize instruction processing in language models that is free from patching assumptions. |
| Outcome: | The proposed method disentangles the implicit linear assumptions of patching-based techniques. |
Split and Rephrase with Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Split and Rephrase (SPRP) tasks require modelling complex grammatical aspects to provide optimal splits and appropriate rephrasing. |
| Approach: | They evaluate large language models on the Split and Rephrase task . they show they can provide large improvements over the state of the art on main metrics . |
| Outcome: | The proposed model outperforms the state-of-the-art model on the Split and Rephrase task on the main metric, but still lacks in splitting compliance. |